UNPKG

25.2 kBHTMLView Raw
1<!DOCTYPE html>
2<html lang="en">
3<head>
4 <meta charset="utf-8">
5 <title>JSDoc: Tutorial: Persistence Adapters</title>
6
7 <script src="scripts/prettify/prettify.js"> </script>
8 <script src="scripts/prettify/lang-css.js"> </script>
9 <!--[if lt IE 9]>
10 <script src="//html5shiv.googlecode.com/svn/trunk/html5.js"></script>
11 <![endif]-->
12 <link type="text/css" rel="stylesheet" href="styles/prettify-tomorrow.css">
13 <link type="text/css" rel="stylesheet" href="styles/jsdoc-default.css">
14</head>
15
16<body>
17
18<div id="main">
19
20 <h1 class="page-title">Tutorial: Persistence Adapters</h1>
21
22 <section>
23
24<header>
25
26
27 <h2>Persistence Adapters</h2>
28</header>
29
30<article>
31 <h1>Overview</h1><p>LokiJS persistence is implemented via an adapter interface. We support autosave and autoload options, simple key/value adapters as well as 'reference mode' adapters, and now internally support various methods of structured serialization which can ease creation of your own persistence adapters, as well as bulk or streamed data exchange. </p>
32<p>An important distinction between an in-memory database like lokijs and traditional database systems is that all documents/records are kept in memory and are not loaded as needed. Persistence is therefore only for saving and restoring the state of this in-memory database.</p>
33<h1>Node.js QuickStart</h1><p>If you are using lokijs in a node environment, we will automatically detect and use the built-in LokiFsAdapter without your needing to provide an adapter.</p>
34<pre class="prettyprint source lang-javascript"><code>var loki = require(&quot;../src/lokijs.js&quot;);
35var db = new loki(&quot;test.db&quot;);</code></pre><p>If you expect your database to grow over 100mb or you experience slow save speeds you might to use our more high-performance LokiFsStructuredAdapter. This adapter utilitizes es6 generator iterators and node streams to stream the database line by line. It will also save each collection into its own file with a file name derived from the base name. This database should scale to support databases just under 1 gb on the default node heap allocation of 1.4gb. Increasing heap allocation, you can push this limit further. </p>
36<p>An example using this LokiFsStructuredAdapter might look like :</p>
37<pre class="prettyprint source lang-javascript"><code>var loki = require('../src/lokijs.js');
38var lfsa = require('../src/loki-fs-structured-adapter.js');
39
40var db = new loki('sandbox.db', { adapter : lfsa});</code></pre><h1>Web QuickStart</h1><p>If you are using lokijs in a web environment, we will automatically use the built-in LokiLocalStorageAdapter. This adapter is limited to around 5gb so that won't last long but here is how to quickly get started experimenting with lokijs :</p>
41<pre class="prettyprint source"><code>&lt;script src=&quot;../../src/lokijs.js&quot;>&lt;/script></code></pre><pre class="prettyprint source lang-javascript"><code>var loki = new loki(&quot;test.db&quot;);</code></pre><p>If you expect your database to grow up to 60megs you might want to use our LokiIndexedAdapter which can save to IndexedDb, if your browser supports it. More information follows on this adapter but here is how to get started quickly with this adapter :</p>
42<pre class="prettyprint source"><code>&lt;script src=&quot;../../src/lokijs.js&quot;>&lt;/script>
43&lt;script src=&quot;../../src/loki-indexed-adapter.js&quot;>&lt;/script></code></pre><pre class="prettyprint source lang-javascript"><code>var idbAdapter = new LokiIndexedAdapter();
44var db = new loki(&quot;test.db&quot;, { adapter: idbAdapter });</code></pre><p>If you expect your database to grow over 60megs things start to get browser dependent. To provide singular guidance and since Chrome is the most popular web browser you will want to employ our LokiPartitioningAdapter in addition to our LokiIndexedAdapter. To sum up as briefly as possible, this will divide collections into their own files and if a collection exceeds 25megs (customizable) it will subdivide into separate pages(files). This allows our indexed db adapter to accomplish a single database save/load using many key/value pairs. This adapter will allow scaling up to around 300mb or so in current testing. </p>
45<p>An example using the LokiPartitioningAdapter along with LokiIndexedAdapter might appear as :</p>
46<pre class="prettyprint source"><code>&lt;script src=&quot;../../src/lokijs.js&quot;>&lt;/script>
47&lt;script src=&quot;../../src/loki-indexed-adapter.js&quot;>&lt;/script></code></pre><pre class="prettyprint source lang-javascript"><code>var idbAdapter = new LokiIndexedAdapter();
48var pa = new loki.LokiPartitioningAdapter(idbAdapter, { paging: true });
49
50var db = new loki('test.db', { adapter: pa });</code></pre><h1>Description of LokiNativescriptAdapter</h1><p>This adapter can be used when developing a nativescript application for iOS or Android, it persists the loki db to the filesystem using the native platform api.</p>
51<h3>Simple Example of using LokiNativescriptAdapter :</h3><pre class="prettyprint source lang-javascript"><code>const loki = require ('lokijs');
52const LokiNativescriptAdapter = require('lokijs/src/loki-nativescript-adapter');
53let db = new loki('loki.json',{
54 adapter:new LokiNativescriptAdapter()
55});</code></pre><blockquote>
56<p>In addition to the above adapters which are included in the lokijs distro, several community members have also created their own adapters using this adapter interface. Some of these include : </p>
57<ul>
58<li>Cordova adapter : https://github.com/cosmith/loki-cordova-fs-adapter</li>
59<li>localForage adapter : https://github.com/paulhovey/loki-localforage-adapter</li>
60</ul>
61</blockquote>
62<h1>Configuring persistence adapters</h1><h2>Autosave, Autoload and close()</h2><p>LokiJS now supports automatic saving at user defined intervals, configured via loki constructor options. This is supported for all persistenceMethods. Data is only saved if changes have occurred since the last save. You can also specify an autoload to immediately load a saved database during new loki construction. If you need to process anything on load completion you can also specify your own autoloadCallback. Finally, in an autosave scenario, if the user wants to exit or is notified of leaving the webpage (window.onbeforeunload) you can call close() on the database which will perform a final save (if needed).</p>
63<blockquote>
64<p><em><strong>Note : the ability of loki to 'flush' data on events such as a browsers onbeforeunload event, depends on the storage adapter being synchronous. Local storage and file system adapters are synchronous but indexeddb is asynchronous and cannot save when triggered from db.close() in an onbeforeunload event. The mouseleave event may allow enough time to perform a preemptive save.</strong></em></p>
65</blockquote>
66<h3>Autosave example</h3><pre class="prettyprint source lang-javascript"><code> var idbAdapter = new LokiIndexedAdapter('loki');
67 var db = new loki('test',
68 {
69 autosave: true,
70 autosaveInterval: 10000, // 10 seconds
71 adapter: idbAdapter
72 });</code></pre><h3>Autosave with autoload example</h3><pre class="prettyprint source lang-javascript"><code> var idbAdapter = new lokiIndexedAdapter('loki');
73 var db = new loki('test',
74 {
75 autoload: true,
76 autoloadCallback : loadHandler,
77 autosave: true,
78 autosaveInterval: 10000, // 10 seconds
79 adapter: idbAdapter
80 });
81
82 function loadHandler() {
83 // if database did not exist it will be empty so I will intitialize here
84 var coll = db.getCollection('entries');
85 if (coll === null) {
86 coll = db.addCollection('entries');
87 }
88 }</code></pre><p><a href="https://rawgit.com/techfort/LokiJS/master/examples/sandbox/LokiSandbox.htm#rawgist=https://gist.githubusercontent.com/obeliskos/447edca33d1274dd9a64767d23df56e9/raw/740d3bedc1ed76d3718acd207b6913281a11ed78/autoloadCallback">Try in Loki Sandbox</a>.</p>
89<h1>Creating your own Loki Persistence Adapters</h1><p>Lokijs currently supports two types of database adapters : 'basic', and 'reference' mode adapters. Basic adapters are passed a string to save and return a string when loaded... this is well suited to key/value stores. Reference mode adapters are passed a reference to the database itself where it can save however it wishes to. When loading, reference mode adapters can return an object reference or serialized string. Below we will describe the minimal functionality which lokijs requires, you may want to provide additional adapter functionality for deleting or inspecting its persistence store.</p>
90<h1>Creating your own 'Basic' persistence adapter</h1><pre class="prettyprint source lang-javascript"><code>MyCustomAdapter.prototype.loadDatabase = function(dbname, callback) {
91 // using dbname, load the database from wherever your adapter expects it
92 var serializedDb = localStorage[dbname];
93
94 var success = true; // make your own determinations
95
96 if (success) {
97 callback(serializedDb);
98 }
99 else {
100 callback(new Error(&quot;There was a problem loading the database&quot;));
101 }
102}</code></pre><p>and a saveDatabase example might look like : </p>
103<pre class="prettyprint source lang-javascript"><code>MyCustomAdapter.prototype.saveDatabase = function(dbname, dbstring, callback) {
104 // store the database, for this example to localstorage
105 localStorage[dbname] = dbstring;
106
107 var success = true; // make your own determinations
108 if (success) {
109 callback(null);
110 }
111 else {
112 callback(new Error(&quot;An error was encountered loading &quot; + dbname + &quot; database.&quot;));
113 }
114}</code></pre><h1>Creating your own 'Reference Mode' persistence adapter</h1><p>An additional 'level' of adapter support would be for your adapter to support <strong>'reference'</strong> mode support. This 'reference' mode will allow lokijs to provide your adapter with a reference to a lightweight 'copy' of the database sharing only the collection.data[] document object instances with the original database. You would use this reference to destructure or save however you want to.</p>
115<p>To instruct loki that your adapter supports 'reference' mode, you will need to implement a top level property called 'mode' on your adapter and set it equal to 'reference'. Having done that and configured that adapter to be used, whenever loki wishes to save the database it will instead call out to an exportDatabase() method on your adapter. </p>
116<p>A simple example of an advanced 'reference' mode adapter might look like : </p>
117<pre class="prettyprint source lang-javascript"><code>function YourAdapter() {
118 this.mode = &quot;reference&quot;;
119}
120
121YourAdapter.prototype.exportDatabase = function(dbname, dbref, callback) {
122 this.customSaveLogic(dbref);
123
124 var success = true; // make your own determinations
125
126 if (success) {
127 callback(null);
128 }
129 else {
130 callback(new Error(&quot;some error occurred.&quot;));
131 }
132}
133
134// reference mode uses the same loadDatabase method signature
135YourAdapter.prototype.loadDatabase = function(dbname, callback) {
136 // do some magic to reconstruct a new loki database object instance from wherever
137 var newDatabase = this.customLoadLogic();
138
139 var success = true; // make you own determinations
140
141 // once reconstructed, loki will expect either a serialized response or a Loki object instance to reinflate from
142 if (success) {
143 callback(newSerialized);
144 }
145 else {
146 callback(new Error(&quot;some error&quot;));
147 }
148}</code></pre><h1>LokiPartitioningAdapter</h1><p>This is an adapter for adapters. It wraps around and converts any 'basic' persistence adapter into one that scales nicely to your memory contraints. It can split your database up, saving each collection independently and only if changes have occurred since the last save. Since each collection is saved separately there is lower memory overhead and since only dirty collections are saved there is improved i/o save speeds. </p>
149<blockquote>
150<p>Chrome (using indexedDb) places a restriction on how large a single saved 'chunk' can be, this Partitioning adapter with just partitioning raises that limit from being 'per db' to 'per collection'... when paging is enabled that limit is raised to being 'per document'. Chrome indexedDb limit is somewhere around 30-60megs sized chunks. </p>
151</blockquote>
152<p>An example using partition adapter with our LokiIndexedAdapter might appear such as :</p>
153<pre class="prettyprint source lang-javascript"><code>var idbAdapter = new LokiIndexedAdapter('appAdapter');
154var pa = new loki.LokiPartitioningAdapter(idbAdapter);
155
156var db = new loki('sandbox.db', { adapter: pa });</code></pre><p>If you expect a single collection to grow rather large you may even want to utilize an additional 'paging' mode that this adapter provides. This is useful if you want to limit the size of data sent to the inner persistence adapter. This paging mode was added to accomodate a Chrome limitation on maximum record sizes. An example using paging mode might appear as follows :</p>
157<pre class="prettyprint source lang-javascript"><code>var idbAdapter = new LokiIndexedAdapter('appAdapter');
158var pa = new loki.LokiPartitioningAdapter(idbAdapter, { paging: true });
159
160var db = new loki('sandbox.db', { adapter: pa });</code></pre><p>You can also pass in a pageSize option if you wish to use a page size other than the default 25meg page size.</p>
161<pre class="prettyprint source lang-javascript"><code>// set up adapter to page using 35 meg page size
162var pa = new loki.LokiPartitioningAdapter(idbAdapter, { paging: true, pageSize:35*1024*1024 });</code></pre><h1>LokiMemoryAdapter</h1><p>This 'basic' persistence adapter is only intended for experimenting and testing since it retains its key/value store in memory and will be lost when session is done. This enables us to verify the partitioning adapter works and can be used to mock persistence for unit testing. </p>
163<p>You might access this memory adapter (which is included in the main source file) similarly to the following :</p>
164<pre class="prettyprint source lang-javascript"><code>var mem = new loki.LokiMemoryAdapter();
165var db = new loki('sandbox.db', {adapter: mem});</code></pre><blockquote>
166<p>In order to see LokiPartitioningAdapter used in conjunction with LokiMemoryAdapter you can view this <a href="https://rawgit.com/techfort/LokiJS/master/examples/sandbox/LokiSandbox.htm#rawgist=https://gist.githubusercontent.com/obeliskos/15c1aa87da16cd89b328eb84bbcdf8fa/raw/d91ac3fee212dc5aa96cb05f479d825faa17c1c8/PartitionedMemoryAdapterTest">Loki Sandbox gist</a> in your browser. </p>
167</blockquote>
168<p>What is happening in the gist linked above is that we create an instance of a LokiMemoryAdapter and pass that instance to the LokiPartitioningAdapter. We utilimately pass in the created LokiPartitioningAdapter instance to the database constructor. We then add multiple collections to our database, save it, update one of the collections (causing that collection's 'dirty' flag to be set), and save again. When we examine the output of the script we can view the contents of the memory adapter's internal hash store to see how there are multiple keys for a single database. We can also see that our modified collection (along with the database container itself) was saved again. The database container currently has no 'dirty' flag set but since we remove all collection.data[] object instances from it, it is relatively lightweight.</p>
169<h1>'Rolling your own' structured serialization mechanism</h1><p>In addition to the <a href="https://github.com/techfort/LokiJS/wiki/Changes-API">ChangesAPI</a> which can be utilized to isolate changesets, LokiJS has established several internal utility methods to assist users in developing optimal persistence or transmission of database contents. </p>
170<p>Those mechanisms include the ability to decompose the database into 'partitions' of structured serializations or assembled into a line oriented format (non-partitioned) and either delimited (single delimited string per collection) or non-delimited (array of strings, one per document). These utility methods are located on the Loki object instance itself as the 'serializeDestructured' and 'deserializeDestructured' methods. They can be invoked to create structured json serialization for the entire database, or (if you pass a partition option) it can provide a single partition at a time. Internal loki structured serialization in its current form provides mild memory overhead reduction and increases I/O time if only some collections need to be saved. It may also be useful for other data exchange or synchronization mechanisms. </p>
171<p>In lokijs terminology the partitions of a database include the database container (partition -1) along with each individual collection (partitions 0-n).</p>
172<p>To destructure in various formats you can experiment with the following parameters :</p>
173<pre class="prettyprint source lang-javascript"><code>var result = db.serializeDestructured({
174 partitioned: false,
175 delimited: false
176});</code></pre><p>To destructure a single partition you might use the following syntax and experiment with 'delimited' and 'partition' properties :</p>
177<pre class="prettyprint source lang-javascript"><code>var result = db.serializeDestructured({
178 partitioned: true,
179 partition: 1,
180 delimited: false
181});</code></pre><blockquote>
182<p>To experiment with the various structured serialization formats you can view this <a href="https://rawgit.com/techfort/LokiJS/master/examples/sandbox/LokiSandbox.htm#rawgist=https://gist.githubusercontent.com/obeliskos/98a73205d7fe9746a687634e19a5eb89/raw/3821c9bbb6bae2689b00d16be9fae78dff430e28/destructuring%2520demo">Loki Sandbox gist</a> and try various combinations of 'partitioned' and 'delimited' options (making sure both the serializeDestructured and deserializeDestructured use the same values.</p>
183</blockquote>
184<p>Destructuring (making many smaller json serializations vs one large serialization) does not lower memory overhead but seems to be a little faster. Partitioning can reduce memory overhead if you can dispose of those memory chunks before advancing to the next (which our adapter implementations do). Our 2.0.0 branch which is able to use ES6 language constructs may gain an iterable interface in the future for data exchange or line-by-line streaming.</p>
185<p>If your database is small enough you can use the LokiPartitioningAdapter (with or without paging) along with LokiMemoryAdapter to decompose database into appropriately sized 'chunks' for transmission.</p>
186<h1>Detailed LokiIndexedAdapter Description</h1><p>Our LokiIndexedAdapter is implemented as a 'basic' mode loki persistence adapter. Since this will probably be the default web persistence adapter, this section will overview some of its advanced features. </p>
187<p>It implements persistence by defining an app/key/value database in indexeddb for storing serialized databases (or partitions). The 'app' portion is designated when instantiating the adapter and loki only supplies it key/value pair for storage.</p>
188<h3>Simple Example of using LokiIndexedAdapter (for browser environments) :</h3><pre class="prettyprint source lang-javascript"><code>&lt;script src=&quot;scripts/lokijs/lokijs.js&quot;>&lt;/script>
189&lt;script src=&quot;scripts/lokijs/loki-indexed-adapter.js&quot;>&lt;/script></code></pre><p>...</p>
190<pre class="prettyprint source lang-javascript"><code>var idbAdapter = new LokiIndexedAdapter('finance');
191var db = new loki('test', { adapter: idbAdapter });</code></pre><p>Note the 'finance' in this case represents an 'App' context and the 'test' designates the key (or database name)... the 'value' is the serialized strings representing your database which loki will provide. Advantages include larger storage limits over localstorage, and a catalog based approach where you can store many databases, grouped by an 'App' context. Since indexedDB storage is provided 'per-domain', and on any given domain you might be running several web 'apps' each with its own database(s), this structure allows for organization and expandibility.</p>
192<blockquote>
193<p><em><strong>Note : the 'App' context is an conceptual separation, not a security partition. Security is provided by your web browser, partitioned per-domain within client storage in the browser/system.</strong></em></p>
194</blockquote>
195<h1>Loki Indexed adapter interface</h1><p>In addition to core loadDatabase and saveDatabase methods, the loki Indexed adapter provides the ability to getDatabaseList (for the current app context), deleteDatabase, and getCatalogSummary to retrieve unfiltered list of app/keys along with the size in database. (Note sizes reported may not be Unicode sizes so effective 'size' it may consume might be double that amount as indexeddb saves in Unicode). The loki indexed adapter also is console-friendly... even though indexeddb is highly asynchronous, relying on callbacks, you can omit callbacks for many of its methods and will log results to console instead. This makes experimenting, diagnosing, and maintenance of loki catalog easier to learn and inspect.</p>
196<h3>Full Examples of using loki indexed adapter</h3><pre class="prettyprint source lang-javascript"><code> // Save : will save App/Key/Val as 'finance'/'test'/{serializedDb}
197 // if appContect ('finance' in this example) is omitted, 'loki' will be used
198 var idbAdapter = new lokiIndexedAdapter('finance');
199 var db = new loki('test', { adapter: idbAdapter });
200 var coll = db.addCollection('testColl');
201 coll.insert({test: 'val'});
202 db.saveDatabase(); // could pass callback if needed for async complete
203
204 // Load database
205 var idbAdapter = new LokiIndexedAdapter('finance');
206 var db = new loki('test', { adapter: idbAdapter });
207 db.loadDatabase({}, function(result) {
208 console.log('done');
209 });
210
211 // Get database list
212 var idbAdapter = new LokiIndexedAdapter('finance');
213 idbAdapter.getDatabaseList(function(result) {
214 // result is array of string names for that appcontext ('finance')
215 result.forEach(function(str) {
216 console.log(str);
217 });
218 });
219
220 // Delete database
221 var idbAdapter = new LokiIndexedAdapter('finance');
222 idbAdapter.deleteDatabase('test'); // delete 'finance'/'test' value from catalog
223
224 // Delete database partitions and/or pages
225 // This deletes all partitions or pages derived from this base filename
226 var idbAdapter = new LokiIndexedAdapter('finance');
227 idbAdapter.deleteDatabasePartitions('test');
228
229 // Summary
230 var idbAdapter = new LokiIndexedAdapter('finance');
231 idbAdapter.getCatalogSummary(function(entries) {
232 entries.forEach(function(obj) {
233 console.log(&quot;app : &quot; + obj.app);
234 console.log(&quot;key : &quot; + obj.key);
235 console.log(&quot;size : &quot; + obj.size);
236 });
237 });</code></pre><h3>Examples of using loki Indexed adapter from console</h3><pre class="prettyprint source lang-javascript"><code> // CONSOLE USAGE : if using from console for management/diagnostic, here are a few examples :
238 var adapter = new LokiIndexedAdapter('loki'); // or whatever appContext you want to use
239 adapter.getDatabaseList(); // with no callback passed, this method will log results to console
240 adapter.saveDatabase('UserDatabase', JSON.stringify(myDb));
241 adapter.loadDatabase('UserDatabase'); // will log the serialized db to console
242 adapter.deleteDatabase('UserDatabase');
243 adapter.getCatalogSummary(); // gets list of all keys along with their sizes</code></pre>
244</article>
245
246</section>
247
248</div>
249
250<nav>
251 <h2><a href="index.html">Home</a></h2><h3>Classes</h3><ul><li><a href="Collection.html">Collection</a></li><li><a href="DynamicView.html">DynamicView</a></li><li><a href="Loki.html">Loki</a></li><li><a href="LokiEventEmitter.html">LokiEventEmitter</a></li><li><a href="LokiFsAdapter.html">LokiFsAdapter</a></li><li><a href="LokiFsStructuredAdapter.html">LokiFsStructuredAdapter</a></li><li><a href="LokiIndexedAdapter.html">LokiIndexedAdapter</a></li><li><a href="LokiLocalStorageAdapter.html">LokiLocalStorageAdapter</a></li><li><a href="LokiMemoryAdapter.html">LokiMemoryAdapter</a></li><li><a href="LokiPartitioningAdapter.html">LokiPartitioningAdapter</a></li><li><a href="Resultset.html">Resultset</a></li></ul><h3>Tutorials</h3><ul><li><a href="tutorial-Autoupdating Collections.html">Autoupdating Collections</a></li><li><a href="tutorial-Changes API.html">Changes API</a></li><li><a href="tutorial-Collection Transforms.html">Collection Transforms</a></li><li><a href="tutorial-Indexing and Query performance.html">Indexing and Query performance</a></li><li><a href="tutorial-Loki Angular.html">Loki Angular</a></li><li><a href="tutorial-Persistence Adapters.html">Persistence Adapters</a></li><li><a href="tutorial-Query Examples.html">Query Examples</a></li></ul>
252</nav>
253
254<br class="clear">
255
256<footer>
257 Documentation generated by <a href="https://github.com/jsdoc3/jsdoc">JSDoc 3.4.0</a> on Sun Dec 18 2016 19:39:52 GMT-0500 (Eastern Standard Time)
258</footer>
259
260<script> prettyPrint(); </script>
261<script src="scripts/linenumber.js"> </script>
262</body>
263</html>
\No newline at end of file